Historical and Recent Trends in the Forecasting Literature

Talk for the WARN-D Friday Science Meeting

February 16, 2024

Introduction

Why talk about this topic?

the secret reason:

Ok but really, why should you care?

“The quiet revolution of numerical weather prediction” (Bauer, Thorpe, and Brunet 2015)

Forecasting Competitions

History

Critical statisticians

  • We should be looking for the true model!

  • Maybe you do not know how to model with ARIMA…

  • I suspect it is more likely to depend on the skill of the analyst … these authors are more at home with simple procedures than with Box-Jenkins. (Chatfield)

Makridakis & Hibon (1979)

  • Our Empirical evidence disagrees

  • Of course we do

  • might be useful for Dr Chatfield to read some of the psychological literature quoted in the main paper, and he can then learn a little more about biases

M-Competitions

Competition Year N° Time Series Insights/Novelty
M1 1982 1001
  • Easy methods work well

  • Combining methods works well

  • Changed forecasting forever

M2 1993 29
  • Not really relevant
M3 2000 3003
  • Somewhat simple models work well with modifications
M4 2020 100,000
  • Combination of ML and Stats works well, pure ML not

  • Probabilistic Prediction

M5 2021 42,000
  • Hierarchical Time Series

  • Ensembles + Pure ML works well

M6 2022 100
  • … to be continued

Utility of Competitions

Advantages

  • Empirical evidence
  • Benchmarking
  • Methodological development and cumulative science (Fildes and Ord 2004)

Issues

Model Selection, Uncertainty & Combination

Model selection

The beauty of simple models:

  • Simple models often outperform more complex ones, even on larger data sets (M1, M3)

  • Simple models are important benchmarks, even in more complex settings

Selecting a single model:

  • ignores uncertainty about this selection (cite Kaplan)
  • tends to perform poorly in forecasting competitions

Move away from model selection to model combination

Model combination

Test stuff

this

Ensemble modeling

Met Office UK & Bauer, Thorpe, and Brunet (2015)

Relevance for psychology

  • Model uncertainty is often neglected, both in inferential and predictive modeling (Kaplan 2021)
  • Current predictive literature often seems to neglect model uncertainty
  • Theoretically linked to the complex systems literature

Probabilistic Forecasting

Probabilistic forecasting methods

What is probabilistic forecasting?

  • what is even meant here?
  • show plot by gneiting in the appendix
  • show different forms

Moving towards probabilistic forecasting

Weather forecasting used probabilistic forecasting surprisingly early:

The probability of rain was much smaller than at other times (Dalton, 1793)

Popularized by Epstein 1969 Stochastic dynamic prediction

Communicating uncertainty

https://www.weather.gov/mrx/probeducation

Relevance for psychology

  • Practically linked to decision theory, e.g., for JITAIs

  • Highly relevant in healthcare settings in general

Mixed Models

Extending mixed models

Flexible Mixed Model

From

\[ y = X\beta + Z\upsilon + \epsilon \]

to

\[y = ml_{fixed}(X)+Z\upsilon + \epsilon\]

(Kilian, Ye, and Kelava 2023)

  • not ‘forecasting’ literature, but ML more broadly
  • use of random effects in machine learning has gained attention

Relevance for Psychology

  • Improving on what we already have
  • Current papers: often rather ad-hoc combinations of forecasts
  • Lots of flexibility
  • For example:
    • lasso with random effects
    • random forest/trees with random effects
    • boosting with random effects

Summary

Issues

Methodology

Transfer to Psychology

Takeaways

  1. Forecasting competitions can lead to methodological improvement
  2. Model uncertainty and model combination are integral parts of forecasting
  3. Probabilistic forecasting is important for decision making
  4. Combining predictions is challenging

Me

References

Bauer, Peter, Alan Thorpe, and Gilbert Brunet. 2015. “The Quiet Revolution of Numerical Weather Prediction.” Nature 525 (7567, 7567): 47–55. https://doi.org/10.1038/nature14956.
Boylan, John E., Paul Goodwin, Maryam Mohammadipour, and Aris A. Syntetos. 2015. “Reproducibility in Forecasting Research.” International Journal of Forecasting 31 (1): 79–90. https://doi.org/10.1016/j.ijforecast.2014.05.008.
Fildes, Robert, and Keith Ord. 2004. “Forecasting Competitions: Their Role in Improving Forecasting Practice and Research.” In, edited by Michael P. Clements and David F. Hendry, 1st ed. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470996430.
Kaplan, David. 2021. “On the Quantification of Model Uncertainty: A Bayesian Perspective.” Psychometrika 86 (1): 215–38. https://doi.org/10.1007/s11336-021-09754-5.
Kilian, Pascal, Sangbeak Ye, and Augustin Kelava. 2023. “Mixed Effects in Machine Learning – A Flexible mixedML Framework to Add Random Effects to Supervised Machine Learning Regression.” Transactions on Machine Learning Research.
Koning, Alex J., Philip Hans Franses, Michèle Hibon, and H. O. Stekler. 2005. “The M3 Competition: Statistical Tests of the Results.” International Journal of Forecasting 21 (3): 397–409. https://doi.org/10.1016/j.ijforecast.2004.10.003.
Makridakis, Spyros, Rob J. Hyndman, and Fotios Petropoulos. 2020. “Forecasting in Social Settings: The State of the Art.” International Journal of Forecasting, M4 Competition, 36 (1): 15–28. https://doi.org/10.1016/j.ijforecast.2019.05.011.
Murphy, Allan H. 1998. “The Early History of Probability Forecasts: Some Extensions and Clarifications.” Weather and Forecasting 13 (1): 5–15. https://doi.org/10.1175/1520-0434(1998)013<0005:TEHOPF>2.0.CO;2.
Strobl, Carolin, and Friedrich Leisch. 2022. “Against the ‘One Method Fits All Data Sets’ Philosophy for Comparison Studies in Methodological Research.” Biometrical Journal Advance Online Publication. https://doi.org/10.1002/bimj.202200104.